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Abstract — For most discrete memoryless channels, there does 
not exist a linear code for the channel which uses all of the 
channel's input symbols. Therefore, linearity of the code for such 
channels is a very restrictive condition and there should be a 
loosening of the algebraic structure of the code to a degree that 
the code can admit any channel input alphabet. For any channel 
input alphabet size, there always exists an Abelian group structure 
defined on the alphabet. We investigate the capacity of Abelian 
group codes over discrete memoryless channels and provide lower 
and upper bounds on the capacity. 

I. Introduction 

Approaching information theoretic performance limits of 
communication systems using structured codes has been an 
area of great interest in recent years [0, ||5], lE], ||8], lITTI . 
lfT4l . The earlier attempts to design fast encoding and decoding 
algorithms resulted in injection of algebraic structures to the 
coding scheme so that the channel input alphabets are replaced 
with algebraic fields and encoders are replaced with matrices. 
It is well-known that binary linear codes achieve the capacity 
of binary symmetric channels Q. More generally, it has also 
been shown that g-ary linear codes can achieve the capacity 
of symmetric channels 161 and linear codes can be used 
to compress a source losslessly down to its entropy ifTOl . 
Optimality of linear codes for certain communication problems 
motivates the study of structured codes in general. 

In 1979, Komer-Marton showed that for multi terminal com- 
munication problems, the asymptotic average performance of 
linear code ensembles can be superior to that of the standard 
code ensembles traditionally used in information theory. In 
the recent past, such gains have been shown for a wide class 
of problems llTTI - |[T3l . Hence information-theoretic character- 
izations of performance of such structured code ensembles for 
various communication problems have become important. 

The algebraic structure of the code, however, imposes 
certain restrictions on the encoder Linear codes are highly 
structured and for certain communication problems such codes 
cannot be optimal. Moreover, they can only be constructed 
on alphabets of certain size (prime power). Group codes are 
a class of algebraic-structured codes that are more general 
because we can construct such codes over any alphabet, and 
they have been shown to outperform unstructured codes in 
certain communication settings ifTTI . Group codes were first 
studied by Slepian ifTSl for the Gaussian channel. In ||T|, the 
capacity of group codes for certain classes of channels has 



been computed. Further results on the capacity of group codes 
were established in JS], ||3|. The capacity of group codes over 
a class of channels exhibiting symmetries with respect to the 
action of a finite Abelian group has been investigated in IS). 

In this work, we focus on the point-to-point channel coding 
problem over general discrete memoryless channels. The chan- 
nel input alphabet is equipped with the structure of an Abelian 
group. We characterize the performance of asymptotically 
good Abelian group codes over general discrete memoryless 
channels. In particular, we derive lower and upper bounds 
on the capacity of Abelian group codes for communication 
over such channels. We use a combination of algebraic and 
information-theoretic tools for this task. 

The paper is organized as follows. In section HI] we in- 
troduce our notation and develop the required background. 
Section |III] presents the lower and upper bound on the ca- 
pacity of Abelian group codes. In section HV] we present two 
special cases, namely, linear codes over arbitrary channels and 
arbitrary Abelian group codes over symmetric channels where 
the two bounds match. 

II. Definitions and notation 

7 ) Group Codes: Given a group G, a group code C over G 
with block length n is any subgroup of G" E), ||8|. A shifted 
group code over G, C + w is a group code C shifted by a fixed 
vector V G G". 

2) Source and Channel Models: We consider discrete mem- 
oryless and stationary channels used without feedback. We 
associate two finite sets X and 3^ with the channel. These 
channels can be characterized by a conditional probability 
law W{y\x). The set X admits the structure of a finite 
abelian group G of the same size. The channel is specified 
by [G^y^W). Assuming a perfect source coding block ap- 
plied prior to the channel coding, the source of information 
generates messages over the set {1, 2, . . . , Af} uniformly. 

3) Achievablility and Capacity : A transmission system with 
parameters (n, Af, r) for reliable communication over a given 
channel (G, W) consists of an encoding mapping and a 
decoding mapping e : {l,2,...,Af} — > G", f : G"- ^ 
{1,2,..., M} such that for all m = 1, 2, . . . , A/, 
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Given a channel (G, y, W), the rate R is said to be achievable 
if for all e > and for all sufficiently large n, there 
exists a transmission system for reliable communication with 
parameters (n, M, r) such that — log M > i? — e, r < e. 
If there is no constraint on the encoder, the maximum achiev- 
able rate is called the (Shannon) capacity of the channel 
and is denoted by C|g| which is known to be equal to 
maxp^ I{X;Y). \G\ denotes the cardinality (size) of the set 
G. We use this notation since only the size and not the 
structure of the channel input alphabet determines the quantity 
C|G|- In this paper, the encoder is constrained to be affine 
and therefore the code is a shifted group code. We denote 
the maximum achievable rate of such codes by Cq- If the 
distribution of X is confined to be uniform over G, we define 
C|^| — I{X] Y). The capacity of shifted group codes over H 
which is itself a subgroup of a larger group G is denoted by 
Ch,g- 

4) Typicality: Consider two random variables X and Y 
with joint probability density function px,y{x, y) over X xy. 
Let n be an integer and e a positive real number The sequence 
pair (a;",y") belonging to X'^ x 3^" is said to be jointly e- 
typical with respect to px,Y{x,y) if 
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and none of the pairs {a,b) with px,Y(a,b) = occurs 
in (a;",y"). Here, N{a,b\x",y") counts the number of oc- 
currences of the pair {a,b) in the sequence pair (a;",?/"). 
We denote the set of all jointly e-typical sequences pairs in 

X" X y" by A'^{X,Y). 

Given a sequence x" G X", the set of conditionally e-typical 
sequences is defined as 



(1) 

any function of e such that 



In our notation, 0(e) i: 
limg_j.o 0{e) = 0. 

III. Bounds on the Capacity of Abelian Group 
Codes 

It is a standard fact (see [|9l and for example) that any 
Abelian group G can be decomposed into Xpr groups in the 
form G = '^pi''i for some integers and primes pi for 

i = 1, 2, • ■ • , / with the possibility of repetitions. Define Ri = 
Zp.i-i to get G = Ri- This means that any element g in 

the group G can be represented by an /-tuple (gi, 32, ■ • ■ ,5/) 
where gi G Ri — {0, 1, • ■ • — 1} and this representation 
preserves the group structure of G. Any subgroup H of G can 

/T\ i 0-- 

be represented by iJ = ^i^iPi' Ri- 
A. Lower bound 

Theorem III.l. A lower bound on the Capacity of group 
codes over the group G = ^^^i Rifor a discrete memoryless 
channel (G, y, W) is given by: 
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where wr = Z][=i ^^T""^' Z^*" ^ — ®[=i Ri '^"'^ ^\s\ 
the mutual information between the channel input and output 
when the input distribution is uniform over the subset S of G. 

The subgroup S of H that achieves the maximum value for 
Gj'^j, is called the optimal subchannel corresponding to the 
subgroup H and is denoted hy H* . 

Proof: We construct an ensemble of homomorphic encoders 
over G with block length n and put a uniform distribution 
over the ensemble. Then we calculate the expected average 
probability of error over the ensemble and observe that for 
rates less than Gg, the average probability of error can be 
made arbitrarily small by increasing the block length. 

1) Construction of the ensemble of codes: Let Wi, i = 
1, 2, • ■ • , / be a set of nonnegative rational weights assigned 
to each module Ri such that — 1 ^^'^ /c be a 

nonnegative integer so that Wik is integer for all i. For each 
set of weights, we define an ensemble of codes by taking 
into account all homomorphisms tp -. ^[^i i?^"'''' — > G". It 
is known that the image of a homomorphism is a subgroup 
of the target group ||9]; Therefore any such homomorphism 
defines a group code C over G. We add a random dither v to 
the code to construct a random shifted group code. 

Let m = 1, 2, • • • , Af be the set of messages. Let k be 
large enough so that a unique message representative u{m) 
from the set Rf''^ can be assigned to each message m. 

The encoding rule is given by e(m) ~ Lp{u{m)) + v where (p 
is an arbitrary homomorphism from ^l^i R^^^ to G" and v 
is a random vector in G". 

At the decoder, after receiving the channel output y, decode 
it to the message m if to is the unique message such that 
u{m) and y are jointly e-typical. Otherwise declare error 

The standard generator of the ring Ri = Zpi-s is the 
multiplicative identity of Ri. Define CiK to be the gener- 
ator for the Kl\\ Ri in ^l^^R^^ for « = I, - - , / and 
K = 1, - - - Wik. Then any element a G ©,[^1 i?^'*^ can be rep- 
resented uniquely as a = aiK^iK where UiK G Ri- This 
decomposition will help us characterizing homomorphisms 
from 0[^i Rl'^'' to G". 

Lemma III.2. Any homomorphism (p : ^l^iR^''' G" 
can be represented as (p ~ {pi, p2, • • ■ , (pn) where each pN, 
N ~ I, - - - ,n is given by: 

= ^aiKg^K 

i,K 

for some gf^ 's, i = 1, ■ ■ ■ , /, K = 1, ■ ■ ■ , Wik in G. 

Proof: Follows from standard algebraic arguments. ■ 
The lemma above facilitates the construction of the ensem- 
ble of codes as follows: Take random elements g^ from the 
group G for n = 1, ■ ■ ■ , N, i ^ I, ■ ■ ■ , I and fc = 1, ■ • • , WiK 
and construct the homomorphism p as mentioned in the 
lemma. Also take a random vector v from G" and use the 
encoding rule e{m) = p}{u{m)) + v. 



The rate of the codes in this ensemble is given by: 

R = -}Wi\og\Ri\ ^ -} w^Ti log Pi 

i=l 4=1 

2) Error Analysis: The expected value of the average 
probability of word error is given by: 



Lemma III.5. The average probability of error over the 
ensemble is bounded above by: 



exp2 I -^^X! [(^* ^ 
e I 1=1 

~ — {n - Oi) \ogp, - H{X,\Y) + H{[X,]e, \Y 
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E P{e{rh)=i,Y" ^y\e{m) = x) + 0{e) 

y£A';^{Y\x) xeAl^{X\y) 



Proof: Provided in the appendix. ■ 
Each random variable Xi can be represented by a tuple 
{[Xi]g-, [Xi]e.) where [Xi]g. indicates the coset selection and 
[Xiloi the value selection in the subgroup p^^ Ri of Ri. Note 
that [Xi]g^ and [Xilg^ are independent. We get. 



We need two lemmas to proceed. 

Lemma III.3. For arbitrary messages m and rh and arbitrary 
vectors x,x £ G", define a ~ u(rn) — u(jh) and h = x — x. 
Define 9{m,'rh) ~ {0i,62,--- ,0i) where 6i is the smallest 
number in {0, 1, ■ • ■ ,ri — 1} such that there exists an index 
if G {1, 2, • • • , Wik} with the property aix G p^^^ Ri\p^'^^ Ri. 
Then, 

P{e{rh) = a;|e(m) ^ x) = 

I nLi ifxex+ 0[^i pfRi 

y otherwise. 

Moreover, for a fixed m, let Tg{m) be the set of all rh with 
d{m, m) ~ (01, 02, ■ ■ ■ ,0i), then 



1 

\Te{m)\<X{[{pr'T'' 



i=i 



Proof: Provided in the appendix. ■ 

Lemma III.4. Let y G [V" be an arbitrary channel output 
sequence. For any x G A^l{X\y), we have 



4=1 



r^A-{y) 



< 



■Q 2n[H(X,\Y)-H{lX,]e,)+0(e)] 



where Xi is the ith component of the channel input random 
variable X. i.e. X i — > (Xi, X2, ■ • ■ ,Xi) where Xi G Ri 
and the random variable [Xi]g. takes values from the set of 
cosets of Pi^ Ri in Ri. 

Proof: Provided in the appendix. ■ 
The following lemma presents an upper bound on the 
average probability of error 



cxp2 < [(r, - 0,) \ogp 

{ i=l 

- — in-0,)logp,-H{[X,]0^\[X,]g^,Y) 



Therefore, the probabihty of error can be made arbitrarily 
small if for all 6, 



V —{n - 0,) log Pi 
^-^ n 

4=1 



< Y.^n~0^\ogp,-U{\X,\e^\{X,\e^,Y) 

4=1 

Let X be a uniform random variable over G and let iJ be the 
subgroup of G isomorphic to ® ViRi- The variable X can 
be thought of as a uniform variable over a random coset of 
H in G. Random selection of the coset is due to the random 
dither and we prove in Lemma IIII.7I that the uniformity of 
the distribution over the coset is due to the group structure of 
the code. The variable X can be represented by two random 
variables \X\ii and \X\h where \X\h is uniform over 11 and 
\X\h has a uniform distribution over cosets of ii in G and 
represents the coset selection. The variable \X\h itself can be 
represented by a tuple ([Xije^ , ■ • ■ , \Xi\qj) where the for each 
i the random variable [Xije^ is a uniform variable over Pi'Ri 
and [Xi]e/s are independent from each other and from [X]h. 
The random variable [X]h can also be represented by a tuple 
{[Xi]g-^,--- ,[Xj]gj) where for each i the random variable 
[Xi]g. is a uniform variable over cosets of pf'i?; in Ri and 
[Xi]e/s are independent from each other and from [X]h. 

I{[X]h;Y\[X]h) 

= I {[X,U, - ■ ■ ,[XgA;Y\[X,]g„ - ■ ■ ,[Xg,]) 



J2li[X..]o.,Y\[X,]g,) 

4=1 

/ 

Y,{r^ - logK - H{[X,]gM,]g^ , Y) 



Therefore, the achievabihty condition is equivalent to 

\\0gp,<I{[X]H\Y\[X]H) 



EWik 
— {n 
n 

i=l 



Wherei? = ©[^iP^i?,. 

The rate of the code is given by i? = ^ logPi 
Therefore, this condition is equivalent to 

Define wjy = '^i — — ^ to get 

2^^_^ ufjri log Pi 
1 



i?< — I([X]h;Y\[X]h) 

Wh 



Note that 

p{[X]h = S)I{[X]h;Y\[X]h ^ S) 

s coset of H 

Since this condition must be satisfied for every subgroup H 
of G and the weights Wi are arbitrary, we conclude that the 
rate 

■ V- 1^1 
n = max mm > 7-^7—^ 

is achievable using group codes over G. The weights Wi can 
be represented as Wi = E.^i fc"'.^(^i^-g.) i°gp. ^ 

where w[ = ^"'j ^ . Since X]i=i = 1' '^^e given 
achievable rate region is equivalent to: 



R* 



\H\ 

max mm > t7^^^ 

luu--- ,wi H<G ^ \G\ Wh 

■wi+---+wi=i g coset of 



where wh = J2i=i ''V-^' ""^i- ^^^^e we have replaced w-'s with 
w,'s for simplicity of notation. 

B. Upper bound 

Definition III.l. A subgroup HofG is called maximal for the 
channel (G, y, W) if for all subgroups S of H, C^^, | > ^i^, |. 

Tlieorem III.6. An upper bound on the capacity of group 
codes over the group G = Rifor a memoryless channel 

(G, 3^, W) is given by: 

r<u 

\s\ 

Cg ^ max min max — — 

wi,--- wi „ maximal s coset of H wh 

u;iH rwi — 1 •' 

where wh = Z^Li ^^^TT'^i ^ - ®l=i Pi'^i '^"'^ ^Is] " 
the mutual information between the channel input and output 
when the input distribution is uniform over the subset S of G. 



1) Converse channel coding theorem: Shannon's inverse 
channel coding theorem asserts that for rates R > I{X\Y) 
lossless communication is not possible. For i = 1, 2, ■ • • , n, 
let Xi be the random variable representing the A^th component 
of the codewords and Yi be the corresponding channel output. 
The rate is bounded above by i? < ^ SILi ^i-^i^ ^i)- 
This theorem admits the generalization to the case where the 
single letter distribution of X is constrained by the structure of 
the code. For the case of shifted group codes, the single letter 
distribution of X can only be uniform on cosets of different 
subgroups of the underlying group. 

2} Uniform single letter distribution over cosets: In the case 
of linear codes, the single letter distribution over the channel 
input symbols is confined to be uniform. This holds for group 
codes also; However, for group codes, it can be uniform over 
any subgroup of the channel input alphabet. 

Lemma III.7. For any group code C < G" where G is an 
arbitrary group, uniform multiletter distribution over messages 
induces a uniform single letter distribution over subgroups 
of G. i.e. the components of the channel input sequence are 
uniformly distributed over some subgroup of G that varies for 
different components. 

Proof: Without loss of generality we prove that the 
nth component of the codewords form a subgroup H of 
G and the uniform distribution over codewords induces a 
uniform distribution over H. Let {ci, C2, • ■ • , cm} be the set 
of codewords and let -P[,i,,i](C) ~ {ci„, C2„, ■ • ■ , ci/„} be the 
set of the nth components of the codewords. It has been shown 
in m that P[„ „] (C) is a subgroup of G. Set H = P[n.n] (<C) 
to conclude the first part of the claim. 

Next, we need to show that the single letter distribution over 
H is uniform. Let H = {hi = 0, /i2,-- - ,h\H\}', then the 
lemma claims that the number of occurrences of each hi in 
the sequence ci„,C2n,-"' , CMn is the same. Let C[i „_i] be 
the set of all codewords that are zero at the nth component. 
It is known that C[i „_i] forms a normal subgroup of C and 
C/C[o,„_i] = P[„,„](C) = Hm- Therefore, = 1^1- 

The number of occurrences of hi ~ in the sequence 
ci„,C2„,-- - ,CAin is equal to |C[i,„_i]|. For each h* e H, 
there exists a codeword c* G C ending with h*, and since 
C is a group code, it is closed under addition and therefore 
c* + C[i „_i] is a subset of C. Since the codewords are distinct, 
the set c*+C[i.„_i] contains |C[i^„_i] | codewords ending with 
h* . We conclude that for each h* E H the existence of at 
least |C[i.„_i]| codewords ending with h* is guaranteed. The 



equality 



_|Ci 



\H\ imposes the number of occurrences 



l<D[i.„-iil 

of each h* to be equal to |C[i.„_i]|. i.e. The single letter 
distribution over H is uniform in the nth position. ■ 

Lemma III.8. For any shifted group code C + w over G, uni- 
form multiletter distribution over messages induces a uniform 
single letter distribution over cosets of subgroups of G. 



Proof: 



Proof: Immediate from the previous lemma. 



3) Converse coding appplied to subchannels: Let G = 
®i=i Abelian group and let H = ®[^]^ Pi^^i be an 

arbitrary subgroup of G and let 5* be the optimal subchannel 
corresponding to the subgroup H. Using standard algebraic 
arguments we can show that for any shifted group code C + v 
where C = ©f^^ i?^ and V is an optimal coset selection 
vector, we have 

/ 

C5 = (Cni/") + i;-t; + 0pfi?^ 

Rcs = - log |c n H" \ = - Yin - e,)h \ogp, 

n n ^ — ^ 

i—l 

Define = '•'f^g'"^' then we get i?Cs = ELi 

Lemma III.9. For a maximal subchannel H of the channel 

{G,y,W), Ch^g < cl. 

Proof: Shannon's coverse implies 

1 " 

R<-Y^I{X,,Yi) 

i=l 

where X,'s have uniform distributions over cosets of sub- 
groups of H. Since H is maximal, all of these distributions 
result in a mutual information less than C^^, | . Therefore, the 
average is also less than G^, . Conclude that R < G^, . ■ 
The lemma implies 

C" * 

Therefore, for all maximal subchannels H, R < — - — ^^^^ . 

E,=i -T--^'"> 

This proves the theorem. 

IV. Special cases 

A. Linear Codes 

The capacity of linear codes has been studied in ||2]. We 
show that for the case of linear codes over Fg, the upper and 
lower bounds are tight and are equal to the capacity given 
in (|2l. Let C be a group code over the field Wq for some 
prime number q. Since the only subgroups of are the trivial 
subgroup and the group F^ itself, the lower bound reduces to 
C|p |; And since Fg is maximal in itself, the upper bound 
also reduces to Gj^ | . Therefore the capacity of linear codes 
over Fq is given by Cf, = G^ | = I{X;Y) where X has a 
uniform distribution over the input alphabet. 

B. Symmetric Channels 

For a symmetric channel, uniform input distribution over 
cosets of an arbitrary subgroup H of G results in the same 
mutual information with the channel output; This means all of 
the cosets of H are optimal and we can pick H* = H. The 
lower bound reduces to 

Gn > max min 

wi, ■■■,wi H<G Wh 

wi-\ ^Wj — l 



Since all of the subgroups are maximal for a symmetric 
channel, the lower bound also reduces to the same expression, 
i.e. The capacity of group codes over symmetric channels is 
given by: 

Gn = max min 

H<G Wh 

wi-\ \-Wj — l 

where wh = ELi ^r H ~ ©Lipfi?,. The 

capacity of Abelian group codes over symmetric channels 
given in |l5] coincides with the new result. 

V. Conclusion 

In this paper, we investigated the performance limits of 
Abelian group codes over discrete memoryless channels. Up- 
per and lower bounds on the capacity of such codes has 
been computed and we presented two special cases where 
the bounds match. Our results unify the known results on the 
capacity of structured codes for the point to point channel cod- 
ing problem and states the information theoretic performance 
limits of structured codes based on the algebraic structure of 
the underlying group. 
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VI. Appendix 
A. proof of lemma I///. 3 1 

Let a = u{m) — u{rh) and h = 



a G 



X — X. First assume 
be a random 



r/kxr 



and h G Z^. and let G G 
matrix and v G be a random vector In order to calculate 



the probability P{u{m)G + v = x,u{m)G + v = x) = 
P {aG = h, u{m)G + v = x) we need to count the number 
of solutions of '}2d=i ^i9i ~ ^ where afs are the elements of 
a and g;'s are the rows of the generator matrix G. There are 
r cases that we should take into account. 

• Case 0: There exists an index m such that G 

In this case, the number of choices for G and v is equal to 
pnr(k-i) yvhereas the total number of choices is equal to 
pnr(fc+i) gjjjQg (jjg matrix G and the vector v are chosen 
equilikely, we have: 

P{aG^h,uG + v^ x) = ^,„.(,^,) = ^ 

Let To (to) be the set of all indices to that fall in this 
category. Then we have |To(to)| — [p^Y ~ (p'"^)'''- 

• Case 9 (6 = 1, 2, . . . , r — 1): The conditions is cases 
up to — 1 are not satisfied and there exists an index to 
such that a„i G p^Zpr\p^+^Zp,'. 

In this case, if h ^ (p^Zpr)" there are no choices fore 
G and v. Otherwise, the number of choices for G and 
V is equal to where as the total number of 

choices is equal to ^"'"(^+1). Since the matrix G and the 
vector are chosen equilikely, we have: 

P{aG u''{i)G + w" = x") = 

r p"%"'-<'=-i) _ 1 







„(2r-e) if X ex + [p^'Lpr)'' 

otherwise. 



Let Tg{m) be the set of all indices m that fall in this 
category. Then we have \Tg{in)\ = {p"^^^)^ — [p'^^^^^Y . 
Our original problem can be addressed by using the above 
result for each ring Ri for which we take the matrix Gk.n = 
{gfx) and replace k by Wik. Since the elements of vn and 
gf{^ are chosen uniformly, their components are independent 
across different rings Ri. Therefore, the joint probability is the 
product of probabilities for each ring and the total number of 
such indices is the product of the number of possible indices 
for each ring Ri. Therefore, 

P{e{m) = X, e(jh) — x) = 

I nil if 5 e X + pfR, 

\ otherwise. 
Alternatively, 



P{e{rh) = x\e(rn) ~ x) = 
1 otherwise. 



Moreover, for a fixed to, let Tg{m) be the set of all to with 

6I(to,to) = {01,02,- ■■ ,0i), then 



\Te{m)\=\{[{j^^-'T'\^-Pr'') 



1=1 



= Therefore, 



\Te{m)\<\{[{pr'T'' 

1=1 



This result can also be confirmed by the straightforward 
method. 



B. proof of WL4\ 

Let x,y,z e G" where x ^ {xi, - ■■ , xi), y ^ {yi,-- ■ ,yi) 
and z = [zi, ■ ■ ■ , zj) where Xi, yi, Zi G i?f . For i = 1, • • • , /, 
define 

/ r ^ 



5* = 



nA-{X\y) 



where X is uniform over G and Xi is uniform over Ri. First 
we show that 5 C 5i x • • • S"/. Let z G 5"; Since z G A'^{X\y), 

1 



-N{a,b\z,y) -pxY{a,b) 



< 



\G\-\y\ 



for arbitrary a = (ai , ■ ■ ■ , aj) E G and b ~ {bi, - ■ ■ ,bi) G G. 
We have: 

-N{ai,b\zi,y) ~pxiY{ai,b) 



- V N{a,b\z,y)- V pxY{a,b) 



02,"' jfl/ 
1 



12 1 ■ " i*^/ 



N{a,b\z,y) -~pxY{a,b) 



< \R2\ X • ■ • X \Ri\ X 



|G|-|3^I 



I^i|-I3^l 

We conclude that zi G Al'{Xi\Y). Since z G \®-=iPt'Ri 



we have zi G 



Therefore, zi G 5i. This is true for 
alH = 1, • • • , /. Conclude that 5 C 5i x ■ • • x 5/. Therefore, 

\S\ < l^il X •■• X 15/1 

By Lemma 5 of ifTTI . we have: 

Conclude that 



)+0(e)) 



C. proof of lemma \III.5\ 

The expected value of the average probability of word error 
is given by: 



M 



E{P,„,(err)} = E ]^ E ^(^M = ^) E 

m — 1 xGG'^ rh—1 

E E P{e{m)=S:,Y'' =y\e{m) = x) + 0{e) 

y<^A^(Y\x) x£A^(X\y) 

M M 

= Ei^E^(^M = -)E E E 



M 



Since e is arbitrary let e and therefore 0(e) — !> to get: 
^{Pavg{err)} < ^e^P2 \ ""E ^ ^j)logPi 

n 



M 

m=l xeG" 



rn=l yeA'yiYlx) xeA'yiXly) 



P{e{m) i|e(m) = + 0(e) 

Using Lemma IIII.3I and Lemma IIII.4I we get: 

M{Pavg{err)} < 

E^E^(^M = -)E E E 



M 

m—l x^G^ 



e r7ieTe{m) y£A^{Y\x) 
I 



56 (^+ [® Li p1' R^] ") n^?(./) 

^^EiJ7E^(^M = -)E E E 

e rnGTg{m} yeA';^{Y\x) 



M 

m=l xeG 



,\Y)-H{[X,]o, 



i=lP.. 



y<^A^{Y\x) 



= E]^E^(«M--)E E 

m=l a;eG" S m(^Tg{r, 

i=l 1=1 Pi 

^E]^E^(«M--)E E 

m=l 2-eG" 8 m£Tg{m) 

I -, 

n2n[H(X,|F)-ff([X.](,,)+0(e)] ^ , Q/gN 

niri—Oi) ^ ' 

= EE]^E^(^M = -)n[(^r^r-* 



n 

4=1 



yi[H(X,\Y)-H{[X,]e^) + 0{<i)] 



1 



0(e) 



=^En 

i=l 



(„r,-e,-.w,kr,n[H{X,\Y)^Hi\X,\e,)+0{e)] 1 

Pi 



0(e) 



